IAES International Journal of Artificial Intelligence (IJ-AD 
Vol. 10, No. 2, June 2021, pp. 414~420 
ISSN: 2252-8938, DOL: 10.1159 1/jai.v 10.i12.pp414-420 Oo 414 


Measure the effectiveness of information systems with the naive 
bayes classifier method 


Agung Triayudi!, Sumiati?, Saleh Dwiyatno*, Dentik Karyaningsih’, Susilawati> 
‘Department of Informatic and Communication Technology, Universitas Nasional, Indonesia 
24D epartment of Informatic, Universitas Serang Raya, Indonesia 
3Department of Computer Systems Engineering, Universitas Serang Raya, Indonesia 
*Department of Information System, Universitas Mathla'ul Anwar, Indonesia 


Article Info ABSTRACT 

Article history: Technological advances at this time are developing very fast, information 
systems became the frontline in technological advancements, the need for 

Received Feb 17, 2020 information systems to support jobs is increasingly high. However, its 

Revised Jan 29, 2021 implementation for users does not have a significant impact, so that it needs 

Accepted Mar 10, 2021 to be reviewed and re-evaluated in the use of the information system built. 


The naive bayes classifier method can provide "effective" and "ineffective" 


conclusions and is used as material for evaluation and improvement. The 
Keywords: purpose of this study is to contribute to measuring the effectiveness of the 
information system, to solve problems with the naive bayes classifier method 
approach which has advantages in the process of classifying data and 
predicting data. From the test results three times, training has been conducted 


Classification 
Data training 


Industrial revolution 4.0 using 100 data, accuracy value of 84.82% and error 15.18%. 
Information systems 
Naive bayes classifier This is an open access article under the CC BY-SA license. 


Corresponding Author: 


Agung Triayudi 

Department of Informatic and Communication Technology 

Universitas Nasional 

Jl. Sawo Manila, RT.14/RW.3, Ps. Minggu, Kec. Ps. Minggu, Kota Jakarta Selatan, Jakarta, Indonesia 
Email: agungtriayudi @civitas.unas.ac.id 


1. INTRODUCTION 

The rapid development of information technology affects the progress of a company in optimizing 
performance by creating various information systems [1], [2]. Information system aims to improve service 
quality to the efficiency of work processes starting from presenting information to support operational 
functions to management functions [3]. Changes in the process of completing work from a manual system to 
a modern system where all activities carried out based on electronics known as e-office certainly have many 
disadvantages in its application. Therefore, an evaluation is needed to measure the effectiveness of 
information systems in a company. It can be measured using a decision support system [4], [5]. 

Decision support systems are defined as a system to support the management levels of an 
organization to make decisions in situations of semi-structured problems [6], [7]. This system is not intended 
to automate decision making but provides interactive tools that support decision making processes [8], [9]. 
Naive bayes is an appropriate method used to measure the effectiveness of information systems, and this 
method is popularly used in classification techniques [10]. In the research, naive bayes develops very 
similarly to the multinomial naive bayes tree method, where one of the data mining techniques for the 
classification of raw data from customers [11], [12]. The results showed that the multinomial naive bayes tree 
(MNBTree) approach had an accuracy rate of 16.26% with 145 features, while the accuracy rate with all 
1665 feature datasets was 73.15% [13], [14]. 
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In a previous study, produce a decision support system application with an accuracy level of 81.18% 
[15]. Therefore, this method was chosen because it was considered simple but produced accurate results [16]. 
In other studies discuss the sentiment analysis that has been raised in recent scientific studies. The method 
used is lexicon based and hybrid which is a combination of lexicon based techniques and machine learning. 
In this study, using the hadoop software framework using the naive bayes and naive bayes complement 
algorithms methods. Experiments conducted with 8 million reviews were classified as positive, negative and 
neutral and used various training dataset sizes and around. The performance of the algorithm is measured 
according to the criteria of precision, accuracy, size-F and calls [17], [18]. 

In other research on the effectiveness of information systems, the calculation model and evaluation 
of system effectiveness are based on the analytic hierarchy process (AHP), where the system architecture can 
be evaluated to support the characteristics of the dataset. [19], [20]. The purpose of this study is to develop a 
decision support system to analyze the effectiveness of the implementation of information systems using a 
web-based naive bayes classifier that can help companies to evaluate and determine the appropriate follow- 
up. This study uses 8 assessment attributes which consist of ease of use of the system, benefits of the system, 
speed of completion of work, improvement of work effectiveness, the information generated, speed of 
information presentation, improvement of service quality, the suitability of the system with needs, and 1 
attribute conclusion namely conclusions. From these attributes predictions will be generated in the form of 
conclusions about the effectiveness of information systems based on previous assessment data. 


2. RESEARCH METHOD 
The stages of research that will be carried out in making a decision support system to assess 
information systems using the naive bayes classifier method are shown in Figure 1. 


Research Research Analysis Research 
Mode! Instruments Results Report 
(1) (2) (3) (4) (5) (6) (7) (8) 
Literature Research Model Research Instrument Data Data Interpretation Report 
Review Development Design Development Collection Analysis Writing 
Research Research Data Research 
Program Design Findings 
| at | 


Figure 1. Phase of research 


2.1. System requirements analysis 
1. Data 
The data needed is questionnaire data about the information system that has been filled by the user. 
2. Hardware 
3. Software 
The software used for the design of the system include: 
a. XAMPP version 3.2.2 
b. Notepad++ 


2.2. Data collection 
Data collected in the form of primary data that is a data-assessment questionnaire obtained directly 
from the data source. Besides, used some literature that was secondary data. 


2.2.1. Primary data 

Primary data is taken in the form of 100 questionnaire data consisting of 8 assessment criteria 
written as K1 to K8. Each criterion has 4 members, namely value. Each value has a level from strongly agree 
(4) to strongly disagree (1). The results of the assessment produce conclusions consisting of effective and 
Ineffective. Effective results are obtained based on the average rating of 3.00-4.00. Effective results are based 
on an average rating of <3.00. Effective results are based on an average rating of <3.00. 3 samples of 100 
primary data taken are shown in Table 1. 
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Table 1. Sample of primary data 


Name Kl K2  K3  K4 KS _ K6_ K7_ K8 Conclusion 
Sangapan 3 3 3 3 3 3 4 4 Effective 
Ratna Kusuma 3 4 3 3 4 4 4 3 Effective 

Name Kl K2 K3 K4 KS K6 K7_ K8 Conclusion 

Suri Mulyani 2 3 3 2 3 3 3 3 Ineffective 


2.2.2. Secondary data 

Literature studies are carried out by studying problems related to the object under study, and getting 
appropriate references in applying a method, for example by studying several books, journals, and articles 
related to the problem under study. 


2.3. Naiive bayes classifier 

Naive bayes is a classification method based on teorama bayes [21]. The naive bayes algorithm, 
which assumes that each variable is independent, independent from each other, has no relationship or 
correlation that can affect the results [22]. Predictions based on the bayes theorem general formula in (1) 
[23]: 


P(X|H)P(H) 


P(H|X) =“ 


() 


Once the data is collected, carried out the data processing through data cleansing, data integration, 
data selection, data transformation [24], and the establishment of the dataset that will be used as training data 
and testing data [25], [26]. 


2.4. Analysis of results and conclusions 
The final stage is to analyze the accuracy of the results in assessing the effectiveness of the 
information system, hereinafter referred to as the conclusion. 


3. RESULTS AND ANALYSIS 
3.1. Data preparation 

At this stage, 100 questionnaires were used as training data. Known class of "Effective" as much as 
95 data and class "Not Effective" by 5 data. The data consists of eight attributes and one attribute ranking 
member and the probabilities are shown in Table 2. 


Table 2. Table frequencies (continue) 


No. Attribute Member Frequency 
1. Ease of use of the system Very easy to use 60 
Easy to use 39 
Difficult to use 1 
Very difficult to use 0 
2. Benefits of the system Strongly agree 72 
Agree 28 
Disagree 0 
Strongly disagree 0 
3. Speed of completion of work Very fast 42 
Fast 55 
Not fast 3 
Very fast 0 
4. Increasing the effectiveness of the work Very true 46 
Right 51 
Not true ) 
Very true 0 
5. Information generated Strongly agree 57 
Agree 40 
Disagree 2; 
Strongly disagree 0 
6. Speed of presenting information Strongly agree 41 
Agree 57 
Disagree 2 
Strongly disagree 0 
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Table 2. Table frequencies 


No. Attribute Member Frequency 
Ts Improved quality of service Strongly agree 62 
Agree 38 
Disagree 0 
Strongly disagree 0 
8. The suitability of the system to the needs of Very appropriate 42 
Corresponding 56 
It is not in accordance with 2 
Very appropriate 0 
9. Conclusion Effective 95 
Ineffective B) 


3.2. System testing 

System testing is done in two ways, namely by looking at the accuracy of the system and comparing 
the calculations produced by the system with manual calculations. 
1. Test the accuracy of the system 

System accuracy testing is done to find out how the naive bayes classifier method works to provide 
accuracy and error values [17]. The test was conducted three times on 100 training data with different testing 
data. 

The first test with 100 training data and 25 testing data. 

accuracy value = (21/25) *100% = 84% 

Error value = (4/25) *100% = 16% 

The second test with 100 training data and 40 testing data. 

accuracy value = (34/40) *100% = 85% 

Error value = (6/40) *100% =15% 

The third test with 100 training data and 55 testing data. 

accuracy value = (47/55) *100% = 85.45% 

Error value = (8/55) *100% = 14.55% 
2. Manual calculations 

The second test is done by comparing the results of the system calculation with the results of manual 
calculations shown in Figure 2. 


No Nama K1 K2 K3 K4 K5 K6 K7 K8 
| 1 | Admin | 4 4 3 | 3 | 3 | 3 3 | 4 | 
| 2 | Dinda Audilla | 2 2 3 | 2 | 1 | 2 3 2 | 
| 3 | Dita Pramesti | 4 3 4 | 3 | 4 | 3 4 4 | 
| 4 | Elis Mublishoh | 4 | 4 a | 3 | 3 | a 4 3 | 


Figure 2. Examples of data test 


The stages of testing the manual calculation sample data in Table 3, which are obtained from the 
primary data sample, are carried out by: 
a. Defining variables 


Table 3. Definition of variables 


x Value | Value 2 Value 3 Value 4 
Kl 4 2 4 4 
K2 4 2; 3 4 
K3 3 3 4 3 
K4 3 2 3 3 
K5 3 1 4 3 
K6 3 2 3 3 
K7 3 3 4 4 
K8 4 2 4 3 
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b. Defining the prior probability 

Prior probability that consists of classes effective (H1) and ineffective (HO) based on Table 2 are 
defined as (2) and (3): 

P (H1) / P(X) = 95/100 = 0.95 (2) 

P (H0)/ P (X) = 5/100 = 0.05 (3) 
c. The posterior probability calculation 


The criteria for calculating the probability (X) per class (H) in Table 4. 


Table 4. Posterior probability 


Value | Value 2 Value 3 Value 4 

H, Ho Hi Ho Hy Hy H, Ho 

Kl 0.6315 0 0.0000 0.2 0.6316 0 0.6316 0 

K2 0.7578 0 0.0000 0.2 0.3684 0.8 0.6316 0 
POSTERIOR K3 0.5473 0.6 0.3684 0.8 0.6316 0 0.3684 0.8 
PROBABILITIES K4 0.5157 0.4 0.0000 0.2 0.3684 0.8 0.3684 0.8 
K5 0.3789 0.8 0.0000 0 0.6316 0 0.3684 0.8 
K6 0.5684 0.6 0.0000 0.2 0.3684 0.8 0.3684 0.8 

K7 0.3473 1 0.3684 0.8 0.6316 0 0.6316 0 
K8 0.4421 0 0.0000 0.2 0.6316 0 0.3684 0.8 


d. Final probability calculation 
The final calculated probability by multiplying the previous probability by the posterior probability 
of each class shown in Table 5. 


Table 5. Final probability 


Value | Value 2 Value 3 Value 4 
P(X |H)) 0.004466368 0 0.00502541 0.001710036 
P (H,) /P (X) 0.95 0.95 0.95 0.95 
P (H, |X) 0.00424305 0 0.00477414 0.001624534 
P(X | Ho) 0 0 0 0 
P (Ho) / P (X) 0.05 0.05 0.05 0.05 
P (Ho |X) 0 0 0 0 


From the calculations in Table 6, the P-value (H1 | X) will be compared with P (HO | X). If the value 
of P (H1 | X) > P (HO | X), it can be concluded that the data entered is classified into the Effective class. 
However, if the value of P (H1 | X) < P (HO | X), it can be concluded that the data entered is classified into 
Ineffective classes. The results of comparison of the four assessment samples that have been included as 
shown in Table 6. This result will be compared with the system prediction results as shown in Table 7. The 
system prediction results in Table 6 show that the conclusions from the prediction results are the same as 
manual calculations. 


Table 6. Comparison of the calculation results 
Comparison Conclusion 
Valuel P (Hy |X) > P (Ho| X) EFFECTIVE 
Value 2 P (H; | X) < P (Ho | X) INEFFECTIVE 
Value3 P(H,| X) > P (Hy | X) EFFECTIVE 
Value4_ PH, | X) > PH | X) EFFECTIVE 


Table 7. System calculation results 


No Name Kl K2 K3 K4 K5 K6 K7 K8 Conclusion 
1 Budi 4 4 3 3 3 3 3 4 EFFECTIVE 
2 Andi 2 2 3 2 1 2 3 2 INEFFECTIVE 
3 Rita 4 3 4 3 4 3 4 4 EFFECTIVE 
4 Dinda 4 3 3 3 3 3 3 3 EFFECTIVE 
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4. CONCLUSION 

The results of the system accuracy test conducted three times using 100 training data resulted in an 
accuracy value of 84% for 25 test data, 85% for 40 test data, and a maximum value of 85.45% for 55 test 
data. With a relatively stable value of accuracy and error values and the results of system calculations with 
relatively similar manual calculations, the decision support system is right to be applied to the naive bayes 
classifier method approach to assess the effectiveness of information systems with an average accuracy value 
of 84.82%. 
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